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BACKGROUND 

The  current  DoD  policy  for  Acquisition  Reform  uses  Commercial-Off-The-Shelf  (COTS) 
products  whenever  possible  to  avoid  development  costs  of  new  systems  acquisition.  The  Next 
Generation  Instrumentation  Bus  (NexGenBus)  project  is  currently  evaluating  commercial 
standards  for  a  high-speed  instrumentation  bus  standard.  These  commercial  standards  need  to  be 
tested,  evaluated,  and  possibly  changed  based  on  the  application.  Adapting  a  commercial 
standid  to  make  it  useable  for  a  specific  application  (e.g.  test  instrumentation)  requires  that  the 
standard  be  evaluated  and  tested  to  insure  compliance  and  to  insure  that  a  non-conforming 
variation  to  the  standard  is  not  created. 


PURPOSE 

When  choosing  a  standard  there  are  overlapping  areas  within  the  standard  and  the  application. 
These  overlapping  areas  contain  critical  areas  that  must  be  tested  to  verify  their  compliance  with 
the  standard.  These  critical  areas  frequently  require  simulate  to  effectively  evaluate  their 
effectiveness  without  building  a  complex  lab  set-up.  These  critical  areas  include  new  protocols, 
loading  analysis,  flow  control  and  error  correction.  Simulating  a  protocol  or  introducing  errors 
in  a  simulation  may  be  much  more  cost  effective  than  doing  actual  testing.  This  philosophy 
requires  the  requisite  test  areas  are  done  in  the  lab  but  the  difficult  or  impossible  test  points  are 
done  in  a  simulation.  This  complements  the  lab  testing  and  saves  the  cost  of  expensive  test 
equipment.  However,  since  a  simulation  is  only  as  good  as  the  parameters  that  are  input,  some 
lab  testing  is  required  to  provide  verification  of  the  simulation. 

SYSTEM  ANALYSIS  AND  MODELING  OBJECTIVES 

The  main  objective  of  the  initial  modeling  effort  is  to  analyze  the  point-to-point  fiber  channel 
architecture  to  determine  the  functional  message  throughput  and  latency.  This  is  done  by 
developing  a  Visual  C++  program  for  the  NexGenBus  program.  The  C++  program  uses  the 
driver  obtained  from  the  fibre  channel  PCI  card  vendor  and  adds  the  features  needed  to  verify  the 
initial  model  of  the  NexGenBus  architecture.  The  C++  program  has  the  following  features: 

•  Allows  the  size  of  the  message  to  be  varied 


•  Allows  the  contents  of  the  payload  to  be  varied  (ASCII  characters) 

•  Allows  a  variable  amount  of  time  to  be  added  between  messages 

•  Allows  a  variable  amount  of  messages  to  be  send 

•  Allows  a  continuous  transfer  of  data  with  a  certain  size  (in  bytes) 

•  Saves  the  file  transferred  to  use  for  comparing  to  original  file 


NEXGENBUS  MODEL  REQUIREMENTS  DEFINITION 

This  section  pertains  only  to  those  requirements  that  will  be  demonstrated  by  simulation. 
The  electrical  tests  will  be  completed  in  a  separate  effort  outlined  in  the  NexGenBus  Test  Plan 
for  Fibre  Channel.  The  requirements  for  the  initial  model  are  to  simulate  the  point-to-point  lab 
set-up  and  verify  that  the  simulation  can  duplicate  the  throughput  and  latency.  The  minimum 
requirement  for  data  rate  on  the  NexGenBus  was  determined  to  be  100  Mbps.  However  to 
ensure  the  bus  chosen  will  meet  future  requirements  it  is  desirable  that  the  data  rate  be  as  high  as 
possible.  Fibre  Channel’s  current  maximum  transmission  rate  of  1.062  Gbaud  yields  a 
unencoded  data  rate  of  800  Mbits  per  second.  The  simulations  will  verify  the  throughput 
recorded  in  the  lab  set-up.  The  baseline  model  will  be  a  point  to  point  topology  and  therefore 
there  will  be  no  measure  of  synchronicity.  The  end-to-end  latency  of  the  bus  transmissions  is 
dependent  on  the  topology  therefore,  the  point-to-point  topology  will  experience  the  lowest 
latency.  The  throughput  measurements  are  calculated  using  the  C-M-  program  as  the  time  to 
transmit  divided  by  the  bytes  received. 

When  the  initial  model  is  verified  a  subsequent  model  will  add  capabilities  such  as 
different  topologies,  additional  nodes  and  additional  message  sources.  By  using  this  expanded 
model,  topology  changes  can  be  quickly  simulated  and  message  delays  evaluated.  In  addition, 
when  another  node  or  message  source  is  needed  we  can  easily  add  it  to  the  model  to  determine 
the  best  placement  for  the  node  and  its  effect  on  the  existing  bus. 


COMNET  in  MODELING  TOOL 

COMNET  IQ  (CACI  Products  Company,  La  Jolla  CA.)  is  a  network  modeling  and  simulation 
tool  that  was  used  to  simulate  the  NexGenBus  data  network  and  estimate  the  performance 
characteristics  of  the  NexGenBus  lab  set-up.  The  lab  set-up  is  show  in  Figure  1.  The  model  is 
created  graphically  using  a  Windows  NT  Graphical  User  Interface  (GUI).  Uses  of  the  COMNET 
software  application  include: 

•  peak  loading  studies 

•  network  sizing  at  the  design  stage 

•  resilience  and  contingent  planning 

•  introductions  of  new  hardware  or  applications 

•  evaluating  performance  improvement  options 

•  evaluating  grade  or  class  of  service 

Objects  representing  the  various  hardware  components  are  created  within  the  application.  The 
building  blocks  include  computer  and  communication  nodes,  switches  and  links  with  their 


parameters  edited  to  match  the  characteristics  of  their  real  world  counterparts.  The  traffic 
loading  and  computer  workloads  were  derived  from  the  vendors  supplying  the  hardware.  Each 
node  is  represented  by  a  processor  node  icon  with  identical  characteristics  to  the  testbed 
configurations.  Connected  to  each  processor  node  is  a  network  message  that  originates  or 
responses  to  the  processor  node.  Name,  periodicity,  origin,  and  destination  characterize  the 
messages.  The  two  processor  nodes  are  connected  via  a  fibre  channel  datalink  which  is 
configured  to  match  the  characteristics  of  the  lab  datalink.  Table  1  lists  the  Configuration 
Parameters  for  the  model  components  that  were  derived  from  their  real-world  counterparts. 

Table  1:  COMNET  III  Configuration  Parameters 


Name  Type 

XmitComp  Processor 
Node 


RAM  (MB)  Buffer  Space  (MB)  Processor  Cycle  (usee) 
64  1  >  .015 


RevComp  Processor 
Node 


64 


.015 


Name  Type 

FibreChannel  Pt-to-Pt  Link 
Link 


B.W.  (Kbps)  Frame  Min.  /  Max.  /  O.H.Qbytes) 
1062.5  32  2016  32 


Name  Type  I  AT  (msec.)  Command  Sequence  ACK.  (bytes)  Msg.  Size 

Xmit_Source  App.  .00075  Local  (Trpt)  Set_Up_Frame  32 

Local  (Wait)  Connection_Auth  None 

Local  (Setup)  Xmit_File  Setup  32  various 

Confirm  32 


Name  Type  Command  Sequence  Msg.  Size  (bytes) 

Receive_Source  Application  Local  (Ansr)  Ack_For_Session  32 


Name  Type  Packets  (bytes! 

FXLP  Protocol  2016 

Flow  Control  Rate  Control 

None  None 


O.H.  (bytes)  Retransmissions  (msec.) 
32  500 


NEXGENBUS  LAB  TESTBED 

The  lab  set-up  consists  of  two  PC’s,  each  of  which  is  running  the  Microsoft  NT  4.0  Operating 
System.  The  workstations  are  a  Pentium  11  running  at  333  MHz  and  a  Pentium  running  at  200 
MHz.  They  each  have  64  Mbytes  of  memory  and  large  hard  disks.  They  have  Systran  PCI  (need 
P/N)  cards  installed  in  a  PCI  card  slot.  There  are  copper  wires  (what  type)  connecting  the 


receive  port  on  one  card  to  the  transmit  port  on  the  other  card.  There  is  a  break  out  connector, 
which  is  used  to  add  the  Ancot  Fibre  channel  protocol  analyzer  between  the  two  PC’s. 


TEST  PROCEDURES 

The  throughput  and  transfer  times  for  different  message  sizes  were  measured  in  the  lab  with  the 
C++  program  developed  for  NexGenBus.  Each  message  size  was  transmitted  from  one  PC  and 
recorded  on  the  receiving  computer.  The  C++  program  was  used  to  transmit  1000  messages  of 
each  message  size.  One  thousand  messages  were  used  to  obtain  a  more  precise  number  for  the 
average  transfer  time  and  throughput.  The  message  sizes  were  32768, 65536, 196608, 1000000 
bytes.  Although  larger  message  could  be  transferred  in  the  lab,  it  was  found  that  to  simulate  file 
sizes  larger  than  1,000,000  bytes  the  simulations  ran  for  an  excessive  amount  of  time.  The 
throughput  and  time  to  transfer  were  recorded  for  each  file  transfer.  For  each  of  the  file  transfers 
a  fixed  amount  of  delay  was  inserted  between  the  file  transfers.  This  delay  was  inserted  to  allow 
the  processor  of  each  PC  to  transfer  the  files  to  the  buffer  on  the  PCI  card.  Without  this  delay 
between  file  transfers  the  processor  became  the  bottleneck  in  the  system  and  prevented  the 
datalink  from  being  utilized  to  the  maximum  available.  The  delays  were  inserted  between  each 
different  file  transfer  were  0, 1,2,  3, 4, 6,  8,  10  and  20  milliseconds.  Table  2  lists  the  file 
transfers  sizes,  recorded  throughput,  transfer  time  and  time  between  messages.  The  primary 
measurements  used  in  the  system  analysis  are  throughput  and  latency.  Throughput  is  defined  as 
the  amount  of  bandwidth  allowed  for  a  particular  message;  for  example,  if  a  1000  byte  message 
takes  .001  seconds  to  go  from  one  computer  to  another,  the  allowed  bandwidth  is  reported  as 
1000  bytes/.OOl  or  1  Mbyte/sec  (8  Mbits/sec).  The  latency  is  defined  as  the  time  for  a  message 
to  be  transported  across  the  network  from  one  computer  to  another. 


Figure  1  NEXGENBUS  Initial  Model 


TEST  RESULTS 

Table  2  shows  the  results  of  the  message  transfer  between  the  two  PC’s  as  show  in  Figure  1. 
Column  1  is  the  size  of  the  file  transfer  (number  of  bytes).  Column  2  is  the  throughput 


(bytes/second)  recorded  in  the  file  transfer.  Column  3  is  the  time  (seconds)  to  complete  the  file 
transfer  and  column  4  is  the  amount  of  delay  (milliseconds)  inserted  between  the  individual  file 
transfers.  All  file  transfers  were  done  1000  times  to  achieve  statistical  averages.  The  300  MHz 
PC  is  transferring  in  the  first  set  of  data  and  the  233  MHz  machine  is  the  transferring  machine  in 
the  second  set  of  data.  Currently  the  transmission  delays  on  the  point-to-point  architecture  are 
approximately  10  microsec.  The  NexGenBus  team  is  currently  investigating  how  to  better 
measure  the  latency  with  the  C++  program. 


Table! 
Test  Results 

Transferring  PC;  300  MHz.  64  Mbyte  RAM 


Ms2  Size  (bytes) 

Throuchnutfbvtes/sec) 

Xfer  Timefsec) 

Delavfms) 

32768 

21.3105 

1.57 

0 

32768 

21.162 

1.569 

1 

32768 

45.003 

2.00 

2 

32768 

45.17 

3.00 

3 

32768 

44.87 

4.00 

4 

32768 

44.90 

7.00 

7 

32768 

44.72 

10.12 

10 

32768 

45.02 

20.0 

20 

65536 

23.85 

2.768 

0 

65536 

23.73 

2.768 

1 

65536 

23.72 

2.785 

2 

65536 

47.11 

3.01 

3 

65356 

66.55 

4.00 

4 

65536 

66.39 

5.00 

5 

65536 

67.23 

6.00 

6 

65536 

61.87 

10.06 

10 

65536 

66.71 

20.00 

20 

196608 

29.29 

6.73 

0 

196608 

29.33 

6.725 

2 

196608 

29.31 

6.73 

4 

196608 

29.26 

6.75 

6 

196608 

73.00 

8.00 

8 

196608 

73.75 

10.01 

10 

196608 

73.63 

20.0 

20 

1000000 

44.9 

22.29 

0 

1000000 

44.9 

22.29 

2 

1000000 

44.88 

22.30 

6 

1000000 

44.88 

22.3 

10 

1000000 

44.84 

22.3 

20 

1000000 

67.84 

30.02 

30 

1000000 

67.82 

35.00 

35 

1000000 

67.82 

40.00 

40 

Transferring  PC:  233  MHz.  64MbYteRAM 

Msg  Size  (bytes)  Throughput(bvtes/sec)  Xfer  Time(sec)  Delav(ms) 


32768 

18.16 

1.85 

0 

32768 

18.14 

1.85 

1 

32768 

18.56 

2.05 

2 

32768 

19.04 

3.01 

3 

32768 

19.237 

4.0 

4 

32768 

19.54 

7.0 

7 

32768 

20.51 

10.0 

10 

32768 

23.53 

19.98 

20 

32768 

26.61 

29.90 

30 

32768 

30.0 

39.96 

40 

32768 

29.10 

49.95 

50 

32768 

30.9 

59.94 

60 

65536 

23.73 

2.80 

0 

65536 

23.73 

2.80 

2 

65536 

28.11 

4.00 

4 

65536 

28.22 

6.00 

6 

65536 

28.50 

8.00 

8 

65536 

27.89 

10.01 

10 

65536 

28.24 

19.99 

20 

196608 

30.39 

6.51 

0 

196608 

30.59 

6.46 

4 

196608 

30.48 

6.49 

6 

196608 

42.60 

8.00 

8 

196608 

42.89 

10.0 

10 

196608 

42.50 

19.9 

20 

1000000 

47.14 

21.25 

0 

1000000 

47.18 

21.24 

2 

1000000 

47.12 

21.26 

6 

1000000 

47.13 

21.26 

10 

1000000 

47.13 

21.26 

20 

1000000 

52.18 

29.90 

30 

1000000 

52.77 

34.90 

35 

1000000 

52.79 

39.90 

40 

Simulations  of  Point-to-Point  File  Transfers  with  300  MHz  P/C 

Ann.  Latency  Trans.  Latency  (msec.) 

32768 

44,516,160 

.255 

.013 

65356 

66,627,552 

.510 

.014 

196608 

73,949,937 

1.530 

.015 

1000000 

66,599,416 

10.585 

.015 

Throughput  (bytes/sec) 


Lab  and  Simulation  Throughput  Measurements 


Lab  and  Simulation  Latency  Measurements 


32768  65536  196608  1000000 

Message  Size  (Bytes) 


Lab 

Simulation 


TEST  vs.  SIMULATION  ANALYSIS 

As  can  be  seen  in  the  graphs  the  best  throughput  results  were  obtained  when  the  delays  were 
inserted  between  the  messages.  This  enabled  the  processor  to  complete  the  PCI  bus  transfers 
before  another  file  transfers  was  started  and  therefore  the  link  was  the  bottleneck  not  the  CPU 
The  fibre  channel  link  during  any  of  the  file  transfers  file  was  never  utilized  over  50  %.  This 


verifies  the  link  is  not  the  bottleneck  in  this  system.  The  throughput  measurement  with  no  delay 
between  the  file  transfers  shows  severely  limits  the  throughput.  The  latency  measurements  with 
no  delay  during  the  lab  file  transfers  matches  the  application  delays  predicted  in  the  simulations. 
Therefore,  the  simulation  can  match  the  PC  transfers  with  no  delay  between  file  transfers  which 
results  in  limiting  the  throughput  and  restricts  the  throughput  on  the  link.  The  simulation  can 
also  match  the  results  of  having  the  PC  processor  not  restrict  the  throughput  in  which  case  the 
maximum  throughput  is  realized  and  the  only  delay  is  the  transmission  delay  on  the  link. 


FOLLOW-ON  EXPANDED  CAPABILITY  MODELS 

The  simulations  have  been  verified  with  the  point-to-point  architecture  in  the  lab.  This  initial 
simulation  model  shows  the  results  are  consistent  with  the  lab  results.  Therefore,  the  model  can 
be  expanded  to  include  different  classes  of  service,  different  protocols,  additional  nodes, 
synchronization  and  timing  issues. 


